Critical appraisal in rapid systematic reviews of COVID-19 studies: implementation of the Quality Criteria Checklist (QCC).

In this letter, we briefly describe how we selected and implemented the quality criteria checklist (QCC) as a critical appraisal tool in rapid systematic reviews conducted to inform public health advice, guidance and policy during the COVID-19 pandemic. As these rapid reviews usually included a range of study designs, it was key to identify a single tool that would allow for reliable critical appraisal across most experimental and observational study designs and applicable to a range of topics. After carefully considering a number of existing tools, the QCC was selected as it had good interrater agreement between three reviewers (Fleiss kappa coefficient 0.639) and was found to be easy and fast to apply once familiar with the tool. The QCC consists of 10 questions, with sub-questions to specify how it should be applied to a specific study design. Four of these questions are considered as critical (on selection bias, group comparability, intervention/exposure assessment and outcome assessment) and the rating of a study (high, moderate or low methodological quality) depends on the responses to these four critical questions. Our results suggest that the QCC is an appropriate critical appraisal tool to assess experimental and observational studies within COVID-19 rapid reviews. This study was done at pace during the COVID-19 pandemic; further reliability analyses should be conducted, and more research is needed to validate the QCC across a range of public health topics.

Navigating the Credibility of Web-Based Information During the COVID-19 Pandemic: Using Mnemonics to Empower the Public to Spot Red Flags in Health Information on the Internet.

Misinformation creates challenges for the general public in differentiating truth from fiction in web-based content. During the COVID-19 pandemic, this issue has been amplified due to high volumes of news and changing information. Evidence on misinformation largely focuses on understanding the psychology of misinformation and debunking strategies but neglects to explore critical thinking education for the general public. This viewpoint outlines the science of misinformation and the current resources available to the public. This paper describes the development and theoretical underpinnings of a mnemonic (Conflict of Interest, References, Author, Buzzwords, Scope of Practice [CRABS]) for identifying misinformation in web-based health content. Leveraging evidence-based educational strategies may be a promising approach for empowering the public with the confidence needed to differentiate truth from fiction in an infodemic.

High quality (certainty) evidence changes less often than low-quality evidence, but the magnitude of effect size does not systematically differ between studies with low versus high-quality evidence.

RATIONALE, AIMS, AND OBJECTIVES: It is generally believed that evidence from low quality of evidence generate inaccurate estimates about treatment effects more often than evidence from high (certainty) quality evidence (CoE). As a result, we would expect that (a) estimates of effects of health interventions initially based on high CoE change less frequently than the effects estimated by lower CoE (b) the estimates of magnitude of effect size differ between high and low CoE. Empirical assessment of these foundational principles of evidence-based medicine has been lacking. METHODS: We reviewed the Cochrane Database of Systematic Reviews from January 2016 through May 2021 for pairs of original and updated reviews for change in CoE assessments based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method. We assessed the difference in effect sizes between the original versus updated reviews as a function of change in CoE, which we report as a ratio of odds ratio (ROR). We compared ROR generated in the studies in which CoE changed from very low/low (VL/L) to moderate/high (M/H) versus M/H to VL/L. Heterogeneity and inconsistency were assessed using the tau and I2 statistic. We also assessed the change in precision of effect estimates (by calculating the ratio of standard errors) (seR), and the absolute deviation in estimates of treatment effects (aROR). RESULTS: Four hundred and nineteen pairs of reviews were included of which 414 (207 × 2) informed the CoE appraisal and 384 (192 × 2) the assessment of effect size. We found that CoE originally appraised as VL/L had 2.1 [95% confidence interval (CI): 1.19-4.12; p = 0.0091] times higher odds to be changed in the future studies than M/H CoE. However, the effect size was not different (p = 1) when CoE changed from VL/L → M/H [ROR = 1.02 (95% CI: 0.74-1.39)] compared with M/H → VL/L (ROR = 1.02 [95% CI: 0.44-2.37]). Similar overlap in aROR between the VL/L → M/H versus M/H → VL/L subgroups was observed [median (IQR): 1.12 (1.07-1.57) vs. 1.21 (1.12-2.43)]. We observed large inconsistency across ROR estimates (I2 = 99%). There was larger imprecision in treatment effects when CoE changed from VL/L → M/H (seR = 1.46) than when it changed from M/H → VL/L (seR = 0.72). CONCLUSIONS: We found that low-quality evidence changes more often than high CoE. However, the effect size did not systematically differ between the studies with low versus high CoE. The finding that the effect size did not differ between low and high CoE indicate urgent need to refine current EBM critical appraisal methods.

Ivermectin and the Integrity of Healthcare Evidence During COVID-19.

The COVID-19 pandemic has been characterized by a lack of clear evidence to guide healthcare professionals, the public and policymakers. The resulting uncertainty, coupled with changing guidelines as additional evidence became available, added to the stress and anxiety reported by decision-makers. Research results are key to providing evidence to guide healthcare decisions. Important questions have arisen about whether various interventions are safe and effective. The evidence found guides those making treatment decisions, and influences those selecting interventions for further evaluation in research studies. As the COVID-19 pandemic intensified, the effectiveness and safety of many pharmaceuticals was queried. Ivermectin will be used to explore the ethics of how healthcare evidence must be critically appraised, even, or especially, during a pandemic. This drug is alleged to be effective in treating COVID-19, with various studies and systematic reviews finding supportive evidence. Some of these have now been linked to concerns about fraud or poor research reporting. This article will focus on the scientific literature and how apparently fraudulent studies were published and influenced treatment decisions, on-going research and public health guidelines. Research evidence is critical during emergencies like pandemics, but urgency should not overtake ethical responsibilities to critically appraise (or evaluate) studies as they become available. These responsibilities apply in various ways to editors, peer-reviewers, news media reporters, and those making treatment decisions, including clinicians, policymakers and the general public. While research article authors have the primary ethical responsibility to reject fraudulent or inaccurate claims, the readers of health research must carefully evaluate all publications. To detect and reject fraudulent healthcare claims, readers need critical appraisal skills that match their level of engagement with those articles. The core principles of critical appraisal will be described in the article, and how they can be adapted for different types of readers. Exemplar tools that develop critical appraisal skills will be noted, with reviews of ivermectin's efficacy explored as examples. As stakeholders in healthcare evidence are increasingly able to identify well-conducted and ethical research they will simultaneously be able to spot and reject fraudulent reports and prevent them from influencing healthcare decisions.

Critical appraisal of clinical practice guidelines for the management of COVID-19: protocol for a systematic review.

BACKGROUND: In December 2019, a novel coronavirus, severe acute respiratory syndrome coronavirus 2 was identified as the cause of an acute respiratory disease, coronavirus disease 2019 (COVID-19). Given the lack of validated treatments, there is an urgent need for a high-quality management of COVID-19. Clinical practice guidelines (CPGs) are one tool that healthcare providers may use to enhance patient care. As such, it is necessary that they have access to high-quality evidence-based CPGs upon which they may base decisions regarding the management and use of therapeutic interventions (TI) for COVID-19. The purpose of the proposed study is to assess the quality of CPGs that make management or TI recommendations for COVID-19 using the AGREE II instrument. METHODS: The proposed systematic review will identify CPGs for TI use and/or the management of COVID-19. The MEDLINE, EMBASE, CINAHL, and Web of Science databases, as well as the Guidelines International Network, National Institute for Health and Clinical Excellence, Scottish Intercollegiate Guidelines Network, and the World Health Organization websites, will be searched from December 2019 onwards. The primary outcome of this study is the assessed quality of the CPGs. The quality of eligible CPGs will be assessed using the Appraisal of Guidelines, Research and Evaluation II (AGREE II) instrument. Descriptive statistics will be used to quantify the quality of the CPGs. The secondary outcomes of this study are the types of management and/or TI recommendations made. Inconsistent and duplicate TI and/or management recommendations made between CPGs will be compared across guidelines. To summarize and explain the findings related to the included CPGs, a narrative synthesis will also be provided. DISCUSSION: The results of this study will be of utmost importance to enhancing clinical decision-making among healthcare providers caring for patients with COVID-19. Moreover, the results of this study will be relevant to guideline developers in the creation of CPGs or improvement of existing ones, researchers who want to identify gaps in knowledge, and policy-makers looking to encourage and endorse the adoption of CPGs into clinical practice. The results of this review will be published in a peer-reviewed journal and presented at conferences. SYSTEMATIC REVIEW REGISTRATION: International Prospective Register for Systematic Reviews (PROSPERO)- CRD42020219944.

COVID-19 Information Sources and Health Behaviors During Pregnancy: Results From a Prenatal App-Embedded Survey.

BACKGROUND: Pregnancy is a time of heightened COVID-19 risk. Pregnant individuals' choice of specific protective health behaviors during pregnancy may be affected by information sources. OBJECTIVE: This study examined the association between COVID-19 information sources and engagement in protective health behaviors among a pregnant population in a large academic medical system. METHODS: Pregnant patients completed an app-based questionnaire about their sources of COVID-19 information and engagement in protective health behaviors. The voluntary questionnaire was made available to patients using a pregnancy app as part of their routine prenatal care between April 21 and November 27, 2020. RESULTS: In total, 637 pregnant responders routinely accessed a median of 5 sources for COVID-19 information. The most cited source (79%) was the Centers for Disease Control and Prevention (CDC). Self-reporting evidence-based protective actions was relatively common, although 14% self-reported potentially harmful behaviors to avoid COVID-19 infection. The CDC and other sources were positively associated with engaging in protective behaviors while others (eg, US president Donald Trump) were negatively associated with protective behaviors. Participation in protective behaviors was not associated with refraining from potentially harmful behaviors (P=.93). Moreover, participation in protective behaviors decreased (P=.03) and participation in potentially harmful actions increased (P=.001) over the course of the pandemic. CONCLUSIONS: Pregnant patients were highly engaged in COVID-19-related information-seeking and health behaviors. Clear, targeted, and regular communication from commonly accessed health organizations about which actions may be harmful, in addition to which actions offer protection, may offer needed support to the pregnant population.

Development of a critical appraisal tool for models predicting the impact of 'test, trace, and protect' programmes on COVID-19 transmission.

OBJECTIVES: To develop a critical appraisal tool for non-computational-specialist public health professionals to assess the quality and relevance of modelling studies about Test and Trace (and Protect - TTP) programmes' impact on COVID-19 transmission. STUDY DESIGN: Decision-making tool development. METHODS: Using Tugwell et al.'s 1985 Health Care Effectiveness equation as a conceptual framework, combined with a purposive search of the relevant early modeling literature, we developed six critical appraisal questions for the rapid assessment of modeling studies related to the evaluation of TTP programmes' effectiveness. RESULTS: By applying the critical appraisal tool to selected recent COVID-19 modeling studies, we demonstrate how models can be evaluated using the six questions to evaluate internal and external validity and relevance. CONCLUSIONS: These six critical appraisal questions are able to discriminate between modeling studies of higher and lower quality and relevance to evaluating TTP programmes' impact. However, these questions require independent validation in a larger and systematic sample of relevant modeling studies which have appeared in later stages of the pandemic.

Outcome Reporting Bias in COVID-19 mRNA Vaccine Clinical Trials.

Relative risk reduction and absolute risk reduction measures in the evaluation of clinical trial data are poorly understood by health professionals and the public. The absence of reported absolute risk reduction in COVID-19 vaccine clinical trials can lead to outcome reporting bias that affects the interpretation of vaccine efficacy. The present article uses clinical epidemiologic tools to critically appraise reports of efficacy in Pfzier/BioNTech and Moderna COVID-19 mRNA vaccine clinical trials. Based on data reported by the manufacturer for Pfzier/BioNTech vaccine BNT162b2, this critical appraisal shows: relative risk reduction, 95.1%; 95% CI, 90.0% to 97.6%; p = 0.016; absolute risk reduction, 0.7%; 95% CI, 0.59% to 0.83%; p < 0.000. For the Moderna vaccine mRNA-1273, the appraisal shows: relative risk reduction, 94.1%; 95% CI, 89.1% to 96.8%; p = 0.004; absolute risk reduction, 1.1%; 95% CI, 0.97% to 1.32%; p < 0.000. Unreported absolute risk reduction measures of 0.7% and 1.1% for the Pfzier/BioNTech and Moderna vaccines, respectively, are very much lower than the reported relative risk reduction measures. Reporting absolute risk reduction measures is essential to prevent outcome reporting bias in evaluation of COVID-19 vaccine efficacy.

COVID-19-related medical research: a meta-research and critical appraisal.

BACKGROUND: Since the start of the COVID-19 outbreak, a large number of COVID-19-related papers have been published. However, concerns about the risk of expedited science have been raised. We aimed at reviewing and categorizing COVID-19-related medical research and to critically appraise peer-reviewed original articles. METHODS: The data sources were Pubmed, Cochrane COVID-19 register study, arXiv, medRxiv and bioRxiv, from 01/11/2019 to 01/05/2020. Peer-reviewed and preprints publications related to COVID-19 were included, written in English or Chinese. No limitations were placed on study design. Reviewers screened and categorized studies according to i) publication type, ii) country of publication, and iii) topics covered. Original articles were critically appraised using validated quality assessment tools. RESULTS: Among the 11,452 publications identified, 10,516 met the inclusion criteria, among which 7468 (71.0%) were peer-reviewed articles. Among these, 4190 publications (56.1%) did not include any data or analytics (comprising expert opinion pieces). Overall, the most represented topics were infectious disease (n = 2326, 22.1%), epidemiology (n = 1802, 17.1%), and global health (n = 1602, 15.2%). The top five publishing countries were China (25.8%), United States (22.3%), United Kingdom (8.8%), Italy (8.1%) and India (3.4%). The dynamic of publication showed that the exponential growth of COVID-19 peer-reviewed articles was mainly driven by publications without original data (mean 261.5 articles ± 51.1 per week) as compared with original articles (mean of 69.3 ± 22.3 articles per week). Original articles including patient data accounted for 713 (9.5%) of peer-reviewed studies. A total of 576 original articles (80.8%) showed intermediate to high risk of bias. Last, except for simulation studies that mainly used large-scale open data, the median number of patients enrolled was of 102 (IQR = 37-337). CONCLUSIONS: Since the beginning of the COVID-19 pandemic, the majority of research is composed by publications without original data. Peer-reviewed original articles with data showed a high risk of bias and included a limited number of patients. Together, these findings underscore the urgent need to strike a balance between the velocity and quality of research, and to cautiously consider medical information and clinical applicability in a pressing, pandemic context. SYSTEMATIC REVIEW REGISTRATION: https://osf.io/5zjyx/.

Proliferation of Papers and Preprints During the Coronavirus Disease 2019 Pandemic: Progress or Problems With Peer Review?

The coronavirus disease 2019 (COVID-19) pandemic has spread exponentially throughout the world in a short period, aided by our hyperconnected world including global trade and travel. Unlike previous pandemics, the pace of the spread of the virus has been matched by the pace of publications, not just in traditional journals, but also in preprint servers. Not all publication findings are true, and sifting through the firehose of data has been challenging to peer reviewers, editors, as well as to consumers of the literature, that is, scientists, healthcare workers, and the general public. There has been an equally exponential rise in the public discussion on social media. Rather than decry the pace of change, we suggest the nephrology community should embrace it, making deposition of research into preprint servers the default, encouraging prepublication peer review more widely of such preprint studies, and harnessing social media tools to make these actions easier and seamless.